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1.  INTRODUCTION 


Technical  studies  provide  insight  to  leaders  that  help  them  make  decisions.  These  decisions  can 
be  of  small  magnitude,  or  could  impact  the  expenditure  of  billions  of  dollars  and  the  lives  of 
many  individuals.  Following  repeatable,  documented  engineering  processes  to  produce  reliable 
and  timely  technical  studies  can  significantly  enhance  the  quality  of  these  decisions.  This  guide 
captures  the  lessons  learned  and  best  practices  from  studies  conducted  in  the  Naval  Sea  Systems 
Command  (NAVSEA)  since  2004.  Much  information  from  this  guide  was  derived  from  the  Ship 
Design  Manager  Manual  (NAVSEA  2009),  a  draft  Ship  Concept  Design  Handbook  (NAVSEA 
2006)  and  a  predecessor  SEA  05D  memorandum  on  "Ship  Concept  Study  Process"  (NAVSEA 
2005).  This  guide  is  consistent  with  the  SEA  05D  memorandum  on  -Surface  Ship  Concept 
Study  Policy”  (NAVSEA  2010).  Mike  Buckley  provided  the  section  on  Decision  Oriented 
Systems  Engineering. 


The  guide  was  written  for  NAVSEA  and  Warfare  Center  Engineers  and  their  supporting 
contractors  in  planning  and  executing  technical  studies.  This  guide  may  also  prove  useful  to 
other  organizations  since  the  issues  and  best  practices  are  not  unique  to  NAVSEA. 

In  its  review  of  the  Mobility  Capabilities  and  Requirements  Study  2016,  The  Government 
Accountability  Office  (GAO  2010)  provides  a  good  list  of  -Generally  Accepted  Research 
Standards”  in  table  1  of  the  document.  While  this  guide  is  consistent  with  these  standards,  study 
leaders  should  consider  reviewing  the  table. 


The  general  technical  study  process  described  in  this  document  is  depicted  in  Figure  1 .  While 
this  process  is  generally  applicable  to  technical  studies  ranging  from  a  few  days  in  duration  to  a 
few  years  in  duration,  the  level  of  effort  in  each  activity  may  differ  significantly.  While  tailoring 
of  the  activities  within  each  "box"  of  Figure  1  is  appropriate,  careful  thought  should  be  given 
prior  to  eliminating  any  "box." 
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Figure  1:  Technical  Study  Process 
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Figure  2  shows  an  alternate  view  of  the  technical  study  process.  A  Design  Structure  Matrix 
(DSM)  shows  the  dependency  of  the  different  activities  upon  each  other.  In  this  rendition  of  the 
DSM,  a  1  in  a  cell  indicates  that  the  activity  corresponding  to  that  row  depends  on  the  activity 
corresponding  to  the  column.  The  DSM  clearly  shows  that  the  Technical  Study  Process  can  be 
thought  of  in  terms  of  two  stages:  Work  Definition  (green),  and  Work  Execution  (blue). 
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Work  Definition  Work  Execution 

Figure  2:  Design  Structure  Matrix  (DSM)  view  of  Technical  Study  Process 

The  organization  of  this  document  is  based  on  Figure  1  and  Figure  2.  Emphasis  is  placed  on  the 
Work  Definition  activities  because  experience  has  shown  that  good  planning  is  critical  to  the 
successful  execution  of  a  study.  Furthermore,  the  details  of  the  Work  Execution  phase  are  often 
study  dependent  and  defined  in  the  Study  Guide  which  is  produced  in  the  Work  Definition  phase. 

NOTE:  This  document  refers  frequently  to  "serialized"  documents.  A  "serialized"  document  is 
dated,  has  a  title,  has  a  unique  identifier  (typically  a  serial  number)  associated  with  it,  is 
approved  by  the  organization  as  a  final  version  of  the  document,  and  is  entered  into  the 
organization's  document  /  correspondence  management  system. 
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2.  WORK  DEFINITION 

2.1  Customer  Tasking 

Customer  tasking  can  come  in  many  different  ways.  Large,  complex  studies  typically  are 
initiated  by  letter  or  memorandum  describing  the  study  tasking  and  the  requesting  organization 
point  of  contact.  Generally,  funding  for  these  studies  is  specified  in  separate  funding  documents. 
Less  complex  studies  may  be  requested  by  email,  hand  written  memo,  or  even  orally. 
Experience  has  shown  however,  that  normally  the  originator  of  the  tasking  is  not  experienced  in 
specifying  exactly  what  is  needed.  Often  the  tasking  is  too  detailed  in  certain  areas  and  lacks 
critical  detail  in  other  areas.  Consequently,  before  expending  too  many  resources  on  a  problem 
that  is  not  well  defined,  it  is  best  to  spend  time  to  properly  define  the  problem  in  a  study  guide 
that  is  understandable  to  the  customer,  study  stakeholders  and  the  Study  Team. 

2.2  Study  Guide  Preparation 

Before  charging  off  to  execute  the  study,  the  study  leader  should  take  time  to  ensure  that  he/she 
is  on  the  same  wavelength  with  the  study  stakeholders  and  customer.  It  is  frustrating  to  discover 
misunderstandings  after  completing  lots  of  fruitless  work.  Everyone  needs  to  fully  understand 
who  the  customer  is,  what  questions  he/she  wants  answered,  when  an  answer  is  expected,  and 
what  resources  are  available.  Developing  the  study  guide  will  lead  to  the  alignment  of  the  study 
to  customer  expectations.  At  a  minimum,  the  study  guide  should: 

1.  Identify  the  customer,  tasking  method,  tasking  date,  what  specific  information  the 
customer  wants  and  what  specific  question  the  customer  wants  answered  from  the 
study. 

2.  Identify  the  general  approach  for  conducting  the  study.  State  the  level  of  effort  to 
provide  the  requested  infonnation,  the  degree  of  uncertainty  in  the  requested 
information,  and  how  long  the  study  will  take. 

3.  List  key  study  assumptions  that  would  require  re-scoping  of  the  study  if  they  were 
changed. 

4.  List  design  and  analysis  tools  (including  version  identification)  that  are  planned  for 
use  as  well  as  a  high  level  Verification,  Validation,  and  Accreditation  (VV&A) 
assessment  of  these  tools. 

5.  Provide  a  high  level  schedule. 

6.  List  funding  or  other  resources  that  your  customer  and  others  will  provide. 

7.  List  major  deliverables  and  when  they  will  be  completed.  Where  necessary,  provide 
some  level  of  detail  as  to  the  contents  of  the  deliverables. 

8.  Identify  the  individual  who  will  lead  the  study. 

One  cannot  overemphasize  the  importance  of  translating  the  customer  tasking  into  one  or  a  few 
specific  questions  that  if  answered  accurately,  will  satisfy  the  customer.  The  advantages  of 
doing  so  include: 
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a.  Work  can  be  planned  to  focus  specifically  on  answering  the  questions.  Any  proposed 
tasks  that  cannot  be  traced  to  answering  the  specific  questions  can  generally  be  cut. 

b.  The  study  will  usually  take  less  time  and  be  cheaper  because  unneeded  work  is  not 
accomplished.  Speed  in  conducting  a  study  improves  relevance  —  if  quality  does  not  suffer.  If  a 
study  takes  too  long  to  produce  results,  many  associated  decisions  will  already  have  been  made. 
Good  results,  fast,  provide  great  value  to  the  customer.  Providing  the  correct  answer  after  the 
associated  decision  has  been  made  has  far  less  value. 

c.  The  ability  to  differentiate  in-scope  and  new-scope  work  is  greatly  enhanced.  If  proposed 
additional  work  does  not  contribute  to  answering  the  questions,  then  it  is  new-scope  and  treated 
as  either  a  new  study  or  a  formal  change  to  the  current  study. 

For  a  short,  simple  study  that  you  are  conducting  alone,  the  study  guide  should  take  no  more  than 
a  few  hours  to  prepare.  A  study  guide  for  a  long  duration  study  involving  a  dozen  or  more 
individuals  in  several  different  organizations  however,  might  require  several  weeks  because  it 
takes  time  to  assemble  a  design  team  and  to  reach  consensus  on  how  to  approach  the  problem. 
Everyone  on  your  team  should  have  access  to  the  study  guide,  as  should  the  stakeholders.  It 
should  be  a  living  document  throughout  the  course  of  your  study,  because  you  will  probably 
need  to  revise  your  assumptions  and  adjust  your  schedule  as  you  gain  knowledge.  Make  sure 
that  everyone  on  the  team  is  aware  of  study  guide  changes  through  serialized  revisions. 

Study  guides  for  moderate  or  long  duration  studies  should  incorporate  a  signature  page  to  include 
the  study  leader,  customer,  and  key  stakeholders. 

Occasionally,  it  may  be  difficult  to  achieve  consensus  among  the  study  stakeholders  and  the 
customer  on  the  assumptions  or  some  other  element  of  the  study  guide.  In  these  cases,  the  study 
leader  has  several  options: 

a.  The  study  leader  can  recognize  the  impasse  in  the  study  guide,  and  provide  a  means 
for  resolving  the  impasse  within  the  context  of  the  study.  For  example,  the  study  could 
incorporate  an  experiment  to  verify  a  contested  assumption. 

b.  If  there  are  only  a  few  relatively  minor  issues  that  are  holding  up  the  issuance  of  the 
study  guide,  then  the  study  leader  may  elect  to  continue  on  with  Work  Definition  and  Work 
Execution  without  an  "approved"  study  guide;  recognizing  that  a  risk  for  rework  exists.  In 
general,  there  should  be  agreement  on  more  than  about  80%  of  the  study  guide  content  before 
proceeding  on  risk. 

c.  If  there  are  significant  issues  holding  up  the  study  guide,  these  issues  should  be 
resolved  by  invoking  the  organization's  processes  for  resolving  technical  issues.  Work  should 
only  be  conducted  to  the  extent  that  the  results  of  the  work  will  resolve  the  issues. 

Studies  can  be  quite  varied,  but  there  are  common  elements  among  successful  planning  efforts. 
Here  are  some  tried  and  true  guidelines. 

Keep  the  objective  in  mind.  It  is  easy  when  planning  or  conducting  a  study  to  get  sidetracked. 
We  all  tend  to  gravitate  towards  technical  areas  that  we  are  more  comfortable  with  or  are 
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more  interested  in.  Make  sure  focus  stays  on  the  right  things.  For  every  step  of  the  plan,  ask 
yourself,  -Will  this  contribute  towards  answering  my  customers  questions?”  And  when  you 
have  finished  your  draft  plan,  ask,  -Will  these  steps  lead  to  the  answers  I  need,  or  have  I 
missed  something?” 

Understand  what  pieces  are  missing.  Every  study  is  a  quest  for  information.  Some  of  it  will  be 
on  hand,  some  you  can  get  with  a  little  research,  and  some  will  take  engineering  to  find.  A 
key  to  managing  study  costs  and  schedules  is  to  take  advantage  of  previous  work  so 
resources  are  not  wasted  in  discovering  the  same  thing  over  and  over.  If  your  study  needs 
new  research  or  tools,  it  is  best  to  figure  that  out  early  since  either  can  take  a  lot  of  time  and 
money  to  complete. 

Dig  to  the  right  depth.  One  of  the  more  challenging  aspects  of  study  planning  is  deciding  how 
deeply  to  investigate  each  technical  area.  Detail  costs  time  and  money,  but  insufficient 
precision  can  cast  doubts  on  the  study4  s  accuracy.  A  good  way  to  address  this  conundrum  is 
to  work  backwards  from  the  technical  areas  that  your  final  report  will  cover,  and  assess  how 
far  you  are  deviating  from  conventional  practice  or  current  technology.  For  example,  if  the 
maximum  weight  for  a  system  planned  for  integration  into  a  ship  is  well  within  the  remaining 
service  life  margin  of  the  ship,  then  a  detailed  weight  estimate  is  likely  not  needed. 
However,  if  the  system  weight  could  exceed  the  remaining  service  life  allowance,  then  more 
detailed  analysis  is  required  to  better  estimate  the  system  weight. 

Parse  the  study  into  logical  chunks.  The  way  to  make  headway  in  a  complex  study  is  to  break  it 
into  manageable  pieces.  Each  activity  should  have  definite  inputs,  objectives  and  products. 

Highlight  task  relationships.  Managing  a  study  team  is  largely  about  making  sure  that  the  right 
data  flows  where  and  when  it  is  needed.  You  must  understand  how  each  of  the  activities 
impacts  the  others,  and  contributes  to  the  overall  objectives.  The  simplest  method  for 
clarifying  task  relationships  is  a  flow  chart.  Another  method  is  the  Design  Structure  Matrix. 
Large  studies  will  benefit  from  program  management  tools  such  as  Microsoft  Project  or 
PLEXUS. 

Be  clear  about  who  will  do  what.  Each  activity  must  have  a  leader  who  understands  what 
product  is  expected,  what  resources  are  available,  and  when  the  activity  is  to  be  completed. 
The  study  leader  might  lead  all  activities  for  simple  short  duration  studies,  or  just  selected 
critical  path  ones  for  larger  efforts. 

Budget  plenty  of  time  for  documentation.  Report  writing  usually  takes  longer  than  expected,  and 
the  process  often  brings  to  light  issues  that  need  further  attention.  A  best  practice  is  to  start 
the  documents  as  early  as  possible  in  the  Work  Execution  phase.  For  large  studies,  an  outline 
of  the  reports  should  be  developed  as  part  of  the  Work  Definition  phase.  Be  warned  that  the 
review  of  documents  by  stakeholders  can  take  longer  than  actually  writing  the  document,  so 
ensure  you  plan  a  sufficient  time  for  document  review. 
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2.3  Prior  Work  Evaluation 

After  making  sure  that  you  understand  the  problem,  the  next  step  is  to  see  if  it  has  already  been 
solved.  The  fastest  way  to  do  this  is  to  ask  other  engineers  within  your  organization.  In 
conducting  a  literature  search,  other  sources  of  historical  study  information  are  your 
organization's  document  management  system,  the  internet,  libraries,  and  the  Defense  Technical 
Information  Center  (DTIC).  Another  term  associated  with  this  activity  is  -Market  Research.” 
Information  on  how  to  conduct  Market  Research  can  be  found  in  POD  1997.  Customers  will 
generally  be  delighted  if  you  quickly  (and  inexpensively)  provide  results  of  a  previous  study  that 
answers  their  questions  directly.  Even  if  your  specific  study  question  hasn‘t  been  addressed  in  a 
previous  study,  you  will  almost  certainly  find  useful  background  data,  similar  studies  for 
comparison,  and  helpful  examples  of  study  methodology. 

2.4  Work  Planning 

2.4.1  Activity  Modeling,  Process  Modeling,  and  Design  Structure  Matrices 

An  important  part  of  planning  a  study  of  even  moderate  complexity  is  modeling  the  study 
process  including  the  activities  comprising  the  study.  In  the  example  shown  in  Figure  3,  study 
activities  are  represented  by  rectangles,  decisions  by  diamonds,  and  the  flow  of  information 
between  study  activities  by  lines.  To  ensure  the  process  is  well  integrated,  particular  attention  is 
needed  to  describe  the  information  expressed  by  the  lines.  The  information  produced  by  the 
source  activity  must  be  compatible  and  of  the  requisite  quality  required  by  the  receiving  activity. 
For  more  information  on  flowcharting,  see  ANSI  X3.6-1970  or  ISO  5807: 1985. 


There  are  many  ways  to  model  a  study  activity  for  integration  into  a  process  model.  A  modeling 
technique  that  has  proven  useful  over  time  is  based  on  the  IDEFO  definition  of  a  function.  As 
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shown  in  Figure  4,  an  activity  interacts  with  external  activities  via  Inputs,  Outputs,  Controls,  and 
Mechanisms.  Inputs  are  those  data  elements  needed  to  perform  the  activity.  Outputs  are  those 
data  elements  that  are  produced  by  the  activity.  Controls  impact  the  manner  in  which  the  activity 
is  performed,  and  Mechanisms  are  the  resources  needed  to  perform  the  activity.  For  further 
information  on  IDEFO,  see  NIST  (1993). 


Controls 


Inputs  ACTIVITY 


Outputs 


Mechanisms 


Figure  4:  IDEFO  Activity  Model 

Within  a  process  model,  an  activity's  boundaries  should  be  chosen  such  that  the  activity  is 
accomplished  by  a  single  organization  using  a  defined  process  and  have  well  defined  artifacts 
(such  as  reports,  data  files,  forms,  test  articles,  etc.)  as  inputs  and  outputs. 

Figure  5  presents  an  alternate  way  to  present  a  process  model.  In  this  Design  Structure  Matrix 
(DSM)  representation,  each  activity  is  represented  by  a  row.  Its  corresponding  column 
represents  -Outputs”  the  activity  produces.  Dots  in  the  off-diagonal  represent  -Inputs”  needed 
by  the  activity.  These  -Inputs”  are  provided  by  other  activities  as  -Outputs”.  If  the  activities  are 
ordered  chronologically  with  the  first  activity  on  top,  then  any  dot  above  the  diagonal  represents 

a  feedback  loop - Inputs”  are  needed  before  they  are  created.  This  input  data  can  either  be 

assumed,  or  the  results  of  a  previous  iteration  can  be  used.  These  "above  the  diagonal"  points 
define  clusters  of  activities  that  are  tightly  coupled.  Consideration  should  be  given  to 
redesigning  the  process  to  eliminate  the  clusters,  co-locating  the  personnel  accomplishing  these 
activities,  and  integrating  design  and  analysis  tools  to  speed  the  execution  of  the  feedback  loop. 

The  DSM  also  can  highlight  activities  that  can  be  accomplished  in  parallel  because  the  two 
activities  do  not  depend  on  each  other. 


Figure  5:  Example  of  a  Design  Structure  Matrix  (DSM)  for  a  Design  Study 
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For  additional  information  on  using  DSM,  see  Doerry  (2009),  Browning  (2001),  Browning 
(2002),  or  the  tutorials  located  on  http://dsmweb.org. 

2.4.2  Study  Cost  and  Schedule  Estimation 

Relatively  few  studies  are  conducted  entirely  with  in-house  resources.  Many  times  support 
contractors  or  Warfare  Center  /  Navy  Research  Laboratory  personnel  are  employed  to  provide 
capability  that  is  not  resident  in-house  or  to  provide  additional  capacity.  The  process  model  is  a 
good  tool  to  estimate  the  time  and  amount  of  labor  required  to  complete  the  study.  Some 
considerations  in  developing  cost  and  schedule  estimates  include: 

-  The  possibility  of  employing  study  participants  part-time  if  constant  workforce  loading 
is  not  achievable.  While  costs  can  be  reduced  by  using  an  engineer  part  time,  schedule  delays 
can  occur  because  of  schedule  conflicts  among  multiple  projects. 

-  The  fully  burdened  cost,  not  the  salary,  of  a  contractor  or  Warfare  Center  /  Navy 
Research  Laboratory  employee  must  be  used  in  calculating  costs.  Additional  funding  for  travel, 
equipment,  and  supplies  may  also  be  needed. 

-  The  anticipated  number  of  iterations  required  to  complete  the  study.  The  DSM  will 
identify  where  iterations  are  needed,  but  without  additional  information,  does  not  provide  insight 
as  to  the  rate  of  convergence  or  the  number  of  iterations  needed. 

-  The  anticipated  involvement  of  stakeholders.  If  stakeholders  are  not  willing  to  engage 
early  in  the  process,  considerable  contingency  funds  /  management  reserve  may  be  needed  to 
develop  unplanned  information  needed  to  satisfy  the  stakeholder. 

-  If  the  study  requires  resources  that  are  limited  in  availability,  the  total  study  time,  and 
therefore  the  costs  can  increase  as  activities  wait  for  missing  data,  or  proceed  at  risk  with  less 
reliable  data. 

-  If  new  tools  or  methods  are  employed  in  the  study,  anticipate  some  rework  as  well  as 
additional  time  for  personnel  to  learn  the  new  tool  or  method. 

-  Do  not  under-estimate  the  cost  and  schedule  impact  of  coordination  meetings,  project 
reviews,  and  peer  reviews.  These  meetings  and  reviews  in  addition  to  other  management 
functions  are  essential  to  producing  a  quality  product,  but  many  times  do  not  appear  in  the 
process  model. 

-  Consider  the  impact  of  holidays  and  vacations  on  schedule.  In  developing  the  schedule, 
consider  assuming  no  work  accomplishment  in  any  week  with  a  holiday,  and  assuming  2  weeks 
unavailable  during  the  summer. 

-  Do  not  under-estimate  the  time  needed  for  the  contracting  process  and  tasking  of  Naval 
Warfare  Centers  /  Navy  Research  Laboratories.  The  schedule  must  account  for  the  estimated 
date  when  different  participants  are  able  to  join  the  study  team. 
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2.4.3  Study  Techniques 

2.4. 3.1  Design  Space  Exploration  &  Design  of  Experiments 

In  many  studies,  the  design  space  under  consideration  is  a  function  of  multiple  interacting 
variables.  In  the  Analysis  of  Alternatives  (AO A)  for  the  Joint  Command  and  Control  Capability 
conducted  in  2000-2001,  the  four  major  drivers  were  staff  size,  survivability,  MSC  vs.  Navy 
Manning,  and  ship  speed.  For  this  study,  these  four  independent  variables  were  fully  explored 
with  24  point  designs  (Figure  6).  The  advantages  of  using  a  design  space  as  compared  to 
conducting  trade-studies  off  of  a  baseline  concept  are: 

-  The  impact  of  an  independent  variable  on  a  dependent  variable  can  be  averaged  over  the  full 
design  space,  not  just  the  two  point  designs  used  in  the  trade-study.  By  averaging  over  the  entire 
design  space,  the  variance  of  the  estimate  can  be  reduced  —  one  produces  better  results. 

-  For  some  studies,  a  baseline  concept  does  not  naturally  exist  —  or  there  is  considerable 
disagreement  among  the  stakeholders  as  to  which  concept  should  be  the  baseline.  A  design 
space  study  doesn't  need  a  baseline  concept,  so  arguments  among  stakeholders  to  establish  one 
are  eliminated.  However,  one  must  ensure  the  design  space  encompasses  the  regions  of  interest 
of  all  stakeholders. 

-  Regions  of  local  and  global  optimums  can  be  better  identified.  In  some  cases,  very  different 
sets  of  independent  variable  values  can  produce  nearly  identical  dependent  variable  values.  A 
simple  trade-study  approach  would  likely  not  identify  this  phenomenon. 
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Greater  than  18.000  m  tons 
15.000  to  18.000  m  tons 
12.000  to  15.000  m  tons 
Less  than  12.000  m  tons 


Figure  6:  Example  of  Design  Space  Study  results  (Doerry  and  Sims  2002) 
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If  the  number  of  important  variables  is  more  than  a  few,  consideration  should  be  given  to  using 
statistical  analysis  tools  such  as  JMP.  Figure  7  and  Figure  8  show  representative  results  from 
JMP.  These  software  tools  also  enable  one  to  use  Design  of  Experiments  techniques  which 
enable  effective  sampling  of  the  design  space  with  fewer  points  than  the  full  sampling  employed 
in  the  JCC(X)  AO  A.  These  alternate  techniques  include  Box-Behnken,  Central  Composite  (Box- 
Wilson)  and  Taguchi  methods  (Whitcomb  2004). 


O  5350  o  O  900  o  fc  29.57 

o  o  ^  in 

°  E(5000,6000)  “  Payload(750,1050)  °  Vs(27,33) 


Figure  7 :  JMP  Prediction  Profiler  (Whitcomb  2004) 


Figure  8:  JMP  Contour  Profiler  (Whitcomb  2004) 


10 


Approved  for  Public  Release 
Distribution  is  Unlimited 


2.4. 3.2  Set  Based  Design  approaches 

Set  Based  Design  (SBD)  is  a  method  where  the  solution  is  obtained  by  systematically 
eliminating  combinations  of  independent  variables  which  can  be  shown  through  study  activities 
to  NOT  be  the  solution.  As  more  and  more  of  the  solution  space  is  eliminated  by  rigorous 
analysis,  the  solution  becomes  apparent.  By  systematically  demonstrating  what  is  not  the 
solution,  study  stakeholder  buy-in  is  generally  easier  to  obtain.  (You  can  diminish  and  address 
the  endless  "Did  you  consider  this?"  questions) 

The  first  step  in  SBD  is  defining  feasible  regions  of  the  solution  space.  This  can  be  either  a 
feasible  variable  range,  such  as  length  or  speed,  or  discrete  states  of  design  such  as  electric  drive 
or  traditional  gear  driven  vessel.  Once  the  feasible  regions  are  established,  different  specialties 
explore  tradeoffs  by  designing/evaluating  multiple  alternatives  within  their  domain.  As  the 
engineers  explore  the  design  alternatives  they  communicate  the  sets  of  possibilities  back  to  the 
other  team  members  and  the  Study  Integrators.  (Singer  et  al.  2009) 

As  depicted  in  Figure  9,  the  three  principle  concepts  to  implement  Set  Based  Design  (SBD)  are 

(1)  consider  a  large  number  of  alternatives  by  understanding  the  solution  space  /  design  space, 

(2)  allow  specialists  to  consider  a  solution  from  their  own  perspective,  and  (3)  use  the 
intersection  between  individual  sets  to  optimize  a  global  solution  and  establish  feasibility  before 
commitment.  (Bernstein  1998).  The  optimization  process  can  consider  physical  perfonnance  of 
a  solution,  as  well  as  other  attributes  such  as  producibility  and  acquisition  complexity. 


Figure  9:  Set  Based  Design  (Bernstein  1998) 


As  each  group  of  specialists  produces  domain  solutions  within  their  area  of  responsibility,  their 
solutions  are  integrated  into  the  larger  context.  To  integrate  by  intersection”  the  Study 
Integrators  lead  the  study  team  in  identifying  intersections  of  feasible  sets  among  the  groups. 
This  requires  prior  agreement  of  the  minimum  and  maximum  bounds  of  each  set.  Specialists 
cannot  extend  beyond  the  bounds  unless  no  other  options  remain. 
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The  ultimate  goal  of  the  integration  process  is  a  smaller  set  of  unified  global  solutions  created  by 
integrating  the  sets  of  domain  solutions  completed  by  different  functional  groups.  The 
integration  process  is  facilitated  by  conceptual  robustness.  Conceptual  robustness  is  achieved 
when  results  concerning  one  aspect  of  a  study  remain  valid  in  the  face  of  subsequent  results  in 
other  aspects  of  the  study.  (Singer  et  al.  2009) 

2.4.33  Decision  Oriented  Systems  Engineering 

One  very  efficient  way  of  looking  at  a  desired  study  is  to  consider  the  study  in  the  light  of  the 
questions  and  decisions  one  intends  to  address  in  conducting  the  study.  What  is  the  ultimate 
question  is  being  asked  and  what  questions  will  have  to  be  asked  to  resolve  the  ultimate 
question?  Such  is  the  decision-oriented  approach  to  defining  and  executing  technical  studies.  If 
the  intended  study  is  not  a  simple  -redo”  with  tweaks  of  an  earlier,  similar  study,  the  decision- 
oriented  approach  can  provide  a  path  to  a  truly  innovative  approach. 

One  decision-oriented  method  known  as  Decision-Oriented  Systems  Engineering  (DOSE) 
involves  looking  at  the  problem  domain  as  an  evolution  of  knowledge  punctuated  by  key 
decisions.  The  premise  behind  the  DOSE  method  is  that  any  problem  (in  this  case,  an 
anticipated  study)  is  best  viewed  as  a  network  of  key  decisions  supported  by  quality  information 
and  human  judgment  that  must  be  successfully  navigated  for  satisfactory  resolution.  This 
evolution  of  knowledge  is  captured  as  a  knowledge  map  based  on  the  conventions  depicted  in 
Figure  10. 


Special  Skill  or 
Knowledge 


DOSE  Knowledge  Map  Conventions 


Required 


ed  "-v 
.  Information 


Figure  10  Conventions  for  a  Completed  DOSE  Knowledge  Map 

The  decisions  are  key  cognitive  decisions  (decisions  involving  human  judgment  or  some 
reasonable  facsimile  thereof),  and  are  described  as  decisions  made,  not  as  enumerated  sets  of 
future  choices.  This  abstraction  allows  for  decisions  to  be  described  as  classes  of  decisions, 
simplifying  the  required  complexity  of  the  decision  network  and  resulting  knowledge  map.  See 
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Figure  1 1  for  an  example  knowledge  map  depicting  a  simplified  and  notional  implementation  of 
Set  Based  Design. 


Figure  11  Notional  DOSE  Knowledge  Map  Depicting  the  Trade  Space  Definition  Element 

of  a  Set-Based  Design  Implementation. 

The  details  of  executing  the  DOSE  method  are  described  in  U.S.  Patent  #7493298  and 
summarized  in  Buckley  &  Stammnitz,  (2004),  and  in  Buckley  &  Womersley,  (2007).  The 
DOSE  method  begins  with  a  partitioning  exercise  and  derived  objective  statements  that  hint  (at 
least)  at  what  will  constitute  successful  navigation  of  the  decisions.  Working  with  appropriate 
subject  matter  experts,  three  artifacts  for  capturing  and  representing  the  evolution  of  knowledge 
are  developed,  beginning  with  the  Key  Decision  Chains,  and  Knowledge  Evolution  Summaries, 
and  culminating  in  the  DOSE  Knowledge  Map.  The  products  of  a  DOSE  analysis  are: 

1.  Key  Decisions:  A  minimum  set  of  key  cognitive  decisions  required  to  navigate  the 
decision  problem  (in  this  case  the  technical  study).  A  key  Decision  is  documented  in  a 
Key  Decision  Product. 

2.  Required  Information  and  Judgment:  A  description  of  the  information  and  human 
judgment  (including  specific  skills  and  expertise)  needed  to  support  those  decisions. 
These  are  indicated  by  "Required  Information,"  "Algorithms/Heuristics,"  "Special  Skills 
or  Knowledge,"  and  "Required  Expertise"  in  Figure  10. 

3.  Essential  Information  Products:  A  minimum  set  of  information  products  and  content 
specifications  (including  human  judgment  needs)  essential  to  successful  completion  of 
the  study.  These  are  indicated  by  "Key  Decision  Product"  in  Figure  10. 

4.  Knowledge  Roadmap:  A  knowledge  map  describing  all  of  the  above. 

5.  Key  Decision  Based  Process  Model:  A  derived  process  model  (for  executing  the  study) 
that  is  based  on  the  knowledge  map,  organized  around  key  decision  milestones,  and 
designed  to  generate  the  essential  information  products. 
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A  few  of  the  advantages  for  the  use  of  decision-oriented  methods  include: 

-  A  study  (process)  punctuated  by  the  key  decisions  needed, 

-  Infonnation  products  that  are  truly  essential  to  the  derived  study  process,  both  of  which 
contribute  to 

-  Simpler,  more  efficient,  derived  study  processes. 

2.4.4  Study  Planning  Considerations 

2. 4.4.1  Study  Phasing 

Breaking  the  scope  of  a  study  into  multiple  phases  may  be  desirable  for  a  number  of  reasons 
including: 

-  Funding  timing  and  origin  may  dictate  multiple  phases 

-  The  scope  or  necessity  of  following  phases  may  not  be  determinable  until  the  results  of  the  first 
phase  are  known. 

-  The  customer  may  need  some  results  faster  than  others.  The  phasing  is  thus  based  on  the 
timing  of  customer  needs. 

-  The  composition  of  the  study  team  may  change  significantly  between  phases  because  of  the 
need  for  different  types  of  and  levels  of  expertise. 

Generally,  study  phases  should  not  be  strictly  serial,  but  should  overlap  somewhat  in  time. 
Essentially,  the  planning  for  a  following  phase  should  occur  during  the  final  execution  of  a 
previous  phase.  This  helps  ensure  there  is  continuity  of  knowledge  or  an  effective  transition  of 
knowledge  from  one  phase  to  another. 

2.4.4.2  Classified,  Sensitive  and  Proprietary  Data 

In  planning  a  study,  active  consideration  must  be  given  to  how  the  teams  will  handle  Classified, 
Sensitive,  and  Proprietary  data.  These  different  classifications  of  data  have  different  rules  with 
respect  to  information  systems  and  the  sharing  of  data.  If  the  use  of  Proprietary  Data  or 
Competition  Sensitive  data  is  anticipated,  non-disclosure  agreements  for  all  non-government 
team  members  are  likely  required.  In  some  cases,  government  team  members  may  also  be 
required  to  sign  a  non-disclosure  agreement. 

The  structure  of  the  final  report  may  also  be  influenced  by  the  classified,  sensitive  and 
proprietary  data.  Many  times,  one  would  like  to  share  the  final  report  with  stakeholders  and 
other  potential  customers;  the  inclusion  of  Classified,  Sensitive,  and  Proprietary  data  may 
preclude  sharing.  Where  possible,  any  data  that  would  unduly  restrict  the  sharing  of  the  final 
report  should  be  included  in  an  appendix  that  can  easily  be  omitted  from  the  report.  Non¬ 
disclosure  agreements  may  be  necessary  to  share  appendices  containing  Proprietary  Data  or 
Competition  Sensitive  data. 
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2.4.43  Dealing  with  Uncertainty 

To  deal  with  uncertainty,  one  must  understand  where  and  how  much  uncertainty  exists  within  the 
study  methods.  One  must  then  understand  how  this  uncertainty  will  impact  the  answers  to  the 
customer's  questions.  This  uncertainty  can  be  a  function  of  the  study  tools  or  processes,  or  could 
be  inherent  in  the  problem  itself.  For  example,  a  given  analysis  tool  may  be  able  to  predict  the 
resistance  of  a  ship  in  waves  accurate  to  10%.  If  this  is  not  adequate,  a  different  tool  or  analysis 
technique  could  be  employed.  On  the  other  hand,  the  precise  price  of  fuel  fifty  years  in  the 
future  is  unknowable.  No  improvement  in  analysis  tools  will  likely  reduce  the  uncertainty  in 
estimated  price. 

Figure  12  is  a  an  example  used  in  the  NAVSEA  Alternate  Propulsion  Study  to  show  the  break¬ 
even  cost  of  crude  oil  where  nuclear  propulsion  becomes  the  more  affordable  solution  for  a  small 
surface  combatant.  This  break-even  cost  is  a  function  of  the  design  features  of  the  ship  as  well 
as  the  manner  in  which  the  ship  is  used  (OP-TEMPO) 


Figure  12:  Example  of  plot  depicted  impact  of  uncertainty  (Webster  et  al.  2007) 

In  planning  the  work  to  accomplish  the  study,  it  is  important  to  understand  what  information  is 
needed  to  generate  the  plots  or  other  means  to  convey  the  customer's  answer.  Knowing  the 
format  of  the  results,  one  can  then  determine  what  activities  are  needed  to  generate  the  results. 

2.4.4.4  Concept  Cost  Estimates 

In  many  studies,  the  customer  will  want  to  understand  cost-perfonnance  trade-offs.  Within 
NAVSEA,  SEA  05C  must  certify  any  potential  acquisition  program  related  cost  estimate 
included  in  a  NAVSEA  document.  Normally  this  means  that  SEA  05C  or  one  of  their 
designated  contractors  should  be  on  the  study  team  to  produce  the  estimates  and  help  guide  the 
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study.  Alternately,  the  study  team  can  have  its  methodology  approved  by  SEA  05C,  then  submit 
the  final  products  to  SEA  05  C  for  final  certification.  The  study  guide  should  include  a 
description  of  the  data  needed  by  SEA  05C  or  other  cost  estimators  to  produce  a  cost  estimate. 
This  data  is  typically  provided  in  the  form  of  a  formal,  serialized  memo  generally  referred  to  as  a 
-Cost  Form.” 

2.4.4. 5  Concept  risk  evaluation  and  readiness  metrics 

In  many  studies,  the  customer  will  want  an  evaluation  of  concept  risk  as  well  as  evaluation  of 
readiness  metrics. 

A  risk  statement  is  generally  in  the  format  of  a  negative  event  such  as  -The  thennal  losses  will 
be  greater  than  can  be  removed  with  the  current  heat  exchanger.”  The  risk  is  evaluated  in  terms 
of  the  severity  of  the  event  and  the  probability  of  the  event.  For  more  information  on  risk 
evaluation,  see  (POD  2006a). 

The  two  readiness  metrics  most  often  evaluated  are  the  Technology  Readiness  Level  (TRL)  and 
Manufacturing  Readiness  Level  (MRL).  Infonnation  on  the  TRL  can  be  found  in  (POD  2009) 
Information  on  the  MRL  can  be  found  in  (POP  2010). 

2.4.4.6  Data  certification 

Many  technical  studies  are  data  driven.  An  important  planning  consideration  is  ensuring  the 
validity  and  applicability  of  data  for  the  intended  use  in  the  study.  All  data  should  be  traceable 
to  a  verified  and  accountable  source.  Pata  that  is  critical  to  the  study  outcome  should  be 
certified  by  the  appropriate  technical  warrant  holder  and/or  program  manager  as  accurate  and 
appropriate  for  use  in  the  study.  This  certification  is  typically  in  the  form  of  a  serialized 
memorandum  from  the  technical  warrant  holder  /  program  manager  to  the  study  leader.  The 
certification  can  also  take  the  form  of  a  Pesign  Pata  Sheet,  standard,  specification,  or  handbook. 

2. 4. 4.7  Organizational  Structure  Considerations 

Many  factors  can  influence  how  a  study  team  is  organized.  In  general,  a  study  leader  should 
strive  to  get  the  right  expertise  on  the  team.  This  may  result  in  the  study  team  being 
geographically  dispersed,  or  composed  of  individuals  from  many  different  organizations.  Small 
studies  may  be  executed  completely  by  a  small  team  of  co-located  individuals  that  divide  up  the 
work  in  an  ad  hoc  fashion.  Larger  studies  however,  require  deliberate  thought  as  to  the  study 
organizational  structure.  Some  basic  principles  in  establishing  an  organizational  structure 
include: 

-  An  activity  should  be  accomplished  by  a  single  co-located  team.  This  team  may  be  assigned 
multiple  activities.  An  individual  may  be  on  multiple  teams. 

-  The  content  of  the  information  passed  between  activities  (and  thus  the  teams)  should  be  well 
defined. 

-  Teams  associated  with  activities  that  form  a  "cluster"  in  the  study  PSM  should  ideally  be  co¬ 
located.  Consideration  should  be  given  to  automating  the  transfer  of  data  among  the  tools  used 
within  the  activities  of  the  "cluster." 
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-  A  study  integrator  or  a  Study  Integration  Team  is  vital  to  ensure  that  the  different  teams  are 
following  the  study  guides,  identifying  and  making  additional  assumptions  that  the  teams  need  to 
accomplish  their  activities,  managing  and  facilitating  the  flow  of  information  among  the 
activities,  and  controlling  the  iterations  within  the  DSM  "clusters."  The  Study  Integration  Team 
should  be  the  authoritative  source  for  any  information  concerning  the  study. 

2.4. 4. 8  Dispersed  Workforce 

It  is  not  unusual  for  a  study  to  have  team  members  located  throughout  the  country  (if  not  the 
world).  This  dispersed  work  force  can  present  some  challenges  (and  opportunities).  Best 
practices  for  working  with  a  dispersed  work  force  include: 

-  Extensive  use  of  video  conferences,  web-based  conferences,  and  teleconferences  for  meetings 
instead  of  travel.  Generally  it  is  good  for  the  first  meeting  and  the  last  meeting  to  be  held  face  to 
face  (to  develop  a  personal  relationship),  but  most  other  meetings  can  be  held  through  video 
conferences  and  teleconferences.  All  meetings  should  be  well  run  by  the  team  leader  (or 
designated  leader)  with  an  agenda  and  meeting  minutes  (including  a  list  of  participants). 

-  Extensive  use  of  a  "shared  directory"  to  enable  teams  to  share  working  documents,  reference 
documents,  and  final  products.  These  three  classes  of  documents  should  be  clearly  partitioned 
from  each  other  within  the  "shared  directory". 

-  Rigorous  publishing  of  meeting  minutes  and  other  documents  on  the  "shared  directory"  to 
ensure  the  entire  workforce  remains  current  with  respect  to  the  progress  of  the  study. 

-  Where  possible,  take  advantage  of  radically  different  time  zones  to  speed  the  iteration  rate  of  a 
"cluster."  For  example  if  a  team  on  the  East  coast  and  a  team  on  the  West  coast  are  part  of  a 
"cluster,"  the  East  coast  team  could  complete  its  activity  by  1  p.m.  and  forward  results  to  the 
West  coast  team  where  the  time  would  be  10  a.m.  The  West  coast  team  could  complete  its 
activity  by  the  close  of  business  and  forward  results  back  to  the  East  coast  team  for  another 
iteration  on  the  following  day.  In  this  manner  the  study  would  accomplish  1 1  hours  of  work  a 
day  without  any  expenditure  of  overtime. 

2.4. 4.9  Validation,  Verification,  and  Accreditation  (VV&A)  of  Design  and  Analysis  Tools 

The  Navy  Modeling  and  Simulation  Office's  "Modeling  and  Simulation  Verification,  Validation, 
and  Accreditation  Implementation  Handbook"  (NMSO  2004)  provides  a  broad,  quick  overview 
of  what  the  VV&A  process  entails. 

Verification.  For  design  and  analysis  tools  the  question  is  "Was  the  tool  built  right?"  Are  the 
software,  data,  assumptions  and  other  tool  attributes  correctly  implemented? 

Validation.  The  process  of  determining  the  degree  to  which  a  tool  is  an  accurate  representation 
of  the  real  world  is  asking  "Was  the  right  tool  built?"  Are  the  physics  based  algorithms  faithful 
to  the  real  world,  are  the  boundary  conditions  correct,  is  the  embedded  data  reliable  and  are  other 
tool  requirements  correct? 

Accreditation.  Accreditation  answers  the  question  "Is  this  the  right  tool  to  use  for  this  particular 
study?" 
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Figure  13  provides  a  simple  pictorial  explanation  of  the  VV&A  terms  defined  above. 


Tests  Evaluates 

“FUNCTIONALITY”  “FIDELITY” 


Determines 

“CREDIBILITY” 


Figure  13:  Verification,  Validation,  and  Accreditation  (NMS0  2004) 


While  most  studies  will  not  be  long  enough  in  duration  to  conduct  formal  Verification  and 
Validation  of  tools,  VV&A  should  be  actively  considered  in  the  selection  of  design  and  analysis 
tools.  The  conscientious  consideration  of  VV&A  provides  credibility  and  confidence  in  the 
results  of  the  study.  VV&A  for  every  design  and  analysis  tool  used  should  be  addressed  in  the 
study  guide  and  in  the  final  report. 


2.4.4.10  Generalizing  Results 

In  many  cases,  the  customer  is  interested  in  the  answer  to  a  generalized  question  that  can  be 
applied  broadly.  Studies  however,  tend  to  look  at  only  a  few  specific  cases.  The  issue  is  how  to 
translate  the  results  of  the  specific  case  to  the  more  general  question.  Key  to  getting  this  right  is 
understanding  the  question,  and  understanding  how  the  answer  will  be  presented  to  the  customer 
and  study  stakeholders.  Both  should  be  accomplished  early  and  documented  in  the  Study  Guide. 

One  technique  that  has  proven  successful  is  to  conduct  a  design  space  exploration  early  using 
simplified  models,  then  calibrate  the  resulting  response  surfaces  with  higher  resolution  models  at 
carefully  chosen  points. 

Where  possible,  use  analytic  based  arguments  to  describe  the  sensitivity  of  point  solutions  to 
changes  in  independent  variables.  The  Customer  is  often  very  interested  (or  should  be 
interested)  in  understanding  how  robust  the  answer  presented  is  to  changes  in  the  assumptions. 


2.4.4.11  Intermediate  Final  Products 

It  is  not  unusual  for  a  customer  to  initiate  a  study  when  he/she  needs  the  information  to  make  a 
decision.  This  means  the  customer  is  interested  in  getting  results  as  fast  as  possible  to  enable  a 
decision  to  be  made.  Hence  the  imperative  to  provide  good  answers  fast.  To  satisfy  this  need  of 
the  customer,  one  should  consider  producing  multiple  final  products  throughout  the  study  period 
to  provide  answers  to  the  customer's  questions  as  soon  as  possible.  Any  final  product  that  is 
delivered  to  the  customer  prior  to  the  commencement  of  the  Study  Closeout  activity  is  an 
intermediate  final  product. 
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Without  intermediate  final  products,  the  tendency  is  to  brief  the  customer  with  intennediate 
results,  then  not  deliver  the  final  product  until  the  end  of  the  study.  Unfortunately  the  customer 
is  often  left  confused  when  the  intennediate  results  change  in  the  final  report.  Providing 
intennediate  final  products  to  the  customer  can  clarify  which  results  are  not  anticipated  to 
change,  and  which  results  require  further  analysis  before  finalizing. 

Another  equally  important  driver  for  Intermediate  final  products  is  gaining  stakeholder  buy-in  to 
the  results.  It  is  always  good  to  know  as  early  as  possible  when  key  stakeholders  are  not  aligned 
with  the  way  the  study  is  progressing.  If  problems  are  identified  early,  the  study  guide  and 
study  process  can  generally  be  slightly  modified  to  address  the  problem.  If  identified  at  the  end 
of  the  study,  time  and  money  will  likely  be  in  short  supply,  and  one  risks  having  a  "non-concur" 
with  the  study  results  from  a  key  stakeholder. 

2.5  Work  Tasking 

A  study  leader  should  be  familiar  with  the  many  ways  to  fund  the  participation  of  contractors  and 
government  engineers  to  accomplish  the  study  activities.  In  developing  tasking  statements,  the 
study  leader  should  consider  the  following: 

-  The  tasking  should  include  the  Activity  definitions  including  inputs,  outputs,  and  the  expected 
activity  duration.  The  tasking  should  also  indicate  the  number  of  iterations  expected. 

-  The  government  should  carefully  consider  what  level  of  Intellectual  Property  Rights  to  retain. 
Inattention  to  these  issues  has  resulted  in  the  past  with  the  government  unable  to  use  the  products 
of  work  they  had  contracted  for.  In  this  case,  the  government  had  to  redevelop  products  using 
in-house  expertise. 

-  The  tasking  should  include  requirements  to  participate  in  integration  meetings,  peer  reviews, 
project  reviews,  and  develop  written  documents. 

-  Some  contractors  and  Government  organizations  have  a  history  of  running  out  of  funds  before 
developing  the  final  report.  This  can  be  alleviated  to  some  degree  by  including  an  option  in  the 
contract  for  follow  on  work  and  making  it  clear  that  the  option  will  only  be  exercised  if  a 
satisfactory  final  report  is  delivered  for  the  base  tasking  in  a  timely  manner.  This  tactic  does 
require  the  study  leader  to  have  an  alternate  plan  to  accomplish  the  follow  on  work  should  the 
contractor  not  deliver  the  final  report. 
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3.  WORK  EXECUTION 

3.1  Project  Management 

3.1.1  Cost,  Schedule  and  Performance 

For  many  small  /  short  duration  studies  where  the  participants  are  generally  the  same  throughout 
the  study,  the  study  leader  should  concentrate  on  getting  the  requisite  level  of  quality  within  the 
allotted  schedule.  If  this  is  achieved,  costs  will  also  likely  be  in  line  with  the  budgeted  amount. 

For  moderate  size  and  duration  studies,  cost,  schedule,  and  performance  must  be  more  closely 
monitored  by  the  study  leader.  The  study  leader  should  use  tools  such  as  Earned  Value 
Management  to  help  identify  potential  problems  and  take  early  corrective  action. 

Large  projects  may  invoke  a  formal  Earned  Value  Management  System  (EVMS)  for 
implementation  throughout  the  project.  Information  on  EVMS  can  be  found  in  POD  (2006). 

Peer  Reviews  can  be  an  effective  means  to  gauge  the  quality  of  the  work  performed  under  the 
study.  The  Project  Reviews  concentrate  more  on  Cost  and  Schedule  Performance. 

3.1.2  Risk  Management 

The  study  leader  should  actively  access  the  risks  associated  with  successfully  completing  the 
study  on  time  and  within  budget.  While  a  formal  risk  management  program  is  likely  only 
desirable  for  the  largest  of  studies,  the  study  leader  should  maintain  a  risk  register,  incorporate 
cost  effective  mitigation  plans  into  the  study  process,  and  maintain  sufficient  contingency  funds  / 
resources  to  cover  the  risks.  See  (POD  2006a)  for  additional  information  on  risk  management. 

3.1.3  Reporting  and  Metrics 

Management  reporting  and  metrics  are  necessary  for  the  effective  execution  of  a  moderate  to 
large  study.  Experience  has  shown  that  a  monthly  status  report  is  not  too  onerous  to  the  teams, 
yet  is  generally  timely  enough  for  study  management  to  understand  the  status  of  the  project. 
These  status  reports  should  include  meaningful  metrics  agreed  to  between  the  study  leader  and 
the  teams  accomplishing  the  activities. 

3.1.4  Scope  Growth  and  Growth  within  Scope 

A  study  leader  should  be  very  careful  in  accepting  growth  in  the  scope  of  work.  The  scope  of 
work  is  generally  bounded  by  the  work  needed  to  answer  the  Customer's  questions  as  articulated 
in  the  study  guide.  Any  additional  work  is  growth  in  scope. 

Where  possible,  the  study  leader  should  work  with  the  customer  to  incorporate  the  increase  in 
scope  in  a  follow  on  phase  to  the  study.  The  need  for  additional  funding  or  resources  should  be 
clearly  identified  in  the  planning  for  the  follow  on  phase. 

Growth  within  scope  is  also  possible.  Growth  within  scope  occurs  when  additional  wok  is 
needed  to  answer  the  Customers  questions,  but  that  work  was  not  directly  budgeted  for.  For  this 
type  of  growth,  the  study  leader  should  strive  to  accommodate  the  additional  work  within  the 
funding  and  resources  provided.  Ideally,  the  study  leader  will  have  included  sufficient 
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contingency  funds  (also  called  -management  reserve”)  in  the  budget  to  accommodate  the 
additional  work.  Resources  can  also  be  reallocated  from  other  tasks,  if  possible.  In  some  cases 
additional  mission-funded  personnel  can  be  applied  to  the  study  to  free  up  funds  for  the 
additional  work. 

If  the  growth  within  scope  is  so  significant  that  the  additional  work  cannot  be  accommodated 
with  the  resources  available  to  the  study  leader,  then  the  study  leader  must  work  with  the 
Customer  to  either  obtain  more  resources  or  reduce  the  scope  of  work. 

3.2  Work  Execution  and  Integration 

Activities  are  completed  in  accordance  with  the  Work  Tasking  and  direction  from  the  Study 
Integrator  /  Study  Integration  Team.  The  Study  Integrator  /  Study  Integration  Team  serves  a 
critical  role  in  ensuring  each  activity  is  provided  with  the  needed  infonnation  and  resources  to 
accomplish  the  activity,  in  resolving  issues  spanning  multiple  activities,  and  in  providing 
guidance  to  individual  activities  to  address  global  study  issues.  Leaders  of  individual  activities 
should  communicate  often  with  the  Study  Integrator  /  Study  Integration  Team  and  with  the 
leaders  of  other  tightly  coupled  (part  of  the  same  DSM  -e-luster”)  activities.  The  Study  Integrator 
/  Study  Integration  Team  should  be  informed  as  quickly  as  possible  of  any  missing  data,  missing 
resources,  missing  assumptions,  or  any  other  issue  that  impacts  the  timely  completion  of  the 
activity  within  cost. 

Careful  attention  must  be  paid  to  ensuring  that  the  individuals  accomplishing  the  study  activities 
are  adhering  to  the  key  assumptions  in  the  study  guide  and  are  producing  the  products  of  the 
requisite  quality  and  in  the  format  required  by  other  study  activities. 

Often,  the  need  to  change  the  study  guide  will  become  apparent  during  Work  Execution  and 
Integration.  Typical  reasons  include  the  unavailability  of  needed  information,  tools  or  methods 
not  working  as  anticipated,  assumptions  found  to  be  inappropriate,  and  planned  resources  not 
being  available.  The  customer  and  stakeholders  should  be  involved  with  any  modification  to  the 
study  guide.  Changes  to  the  study  guide  may  also  result  in  the  need  to  change  work  tasking 
documents. 

3.3  Peer  Review 

Conduct  peer  review(s)  at  appropriate  time(s).  For  many  studies,  peer  reviews  at  the  midpoint 
and  90%  completion  time  (prior  to  the  associated  project  review)  are  appropriate.  The  first 
allows  a  mid-course  guidance  correction  to  the  study  once  initial  data  is  produced,  and  the 
second  hopefully  will  identify  potential  major  mistakes  before  the  study  funding  and  time  runs 
out.  Peers  should  include  subject  matter  experts  that  have  not  been  directly  involved  in  the 
conduct  of  the  study.  Consensus  is  not  the  goal.  The  goal  is  independent  technical  evaluation  of 
the  study  assumptions,  methods,  data,  and  results.  If  consensus  is  not  achieved,  the  dissenting 
inputs  should  be  documented,  and  may  be  appropriate  to  include  in  the  study  report. 

For  short  duration  studies  of  a  few  weeks  or  less,  the  peer  review  may  consist  of  an  informal 
hour  meeting  with  one  or  more  subject  matter  experts.  The  results  could  be  documented  in  a 
manner  such  as  a  memorandum  for  the  record,  or  even  an  email.  For  longer  duration  studies,  the 
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peer  review  could  last  a  half  a  day  with  results  documented  in  formal,  serialized  meeting 
minutes. 

Inviting  the  customer  is  generally  recommended;  however  the  peer  review  should  generally  still 
be  conducted  if  the  customer  is  not  available. 

3.4  Project  Review 

The  number  and  depth  of  your  meetings  and  project  reviews  depends  on  the  duration  and 
complexity  of  your  study.  The  purposes  of  these  meetings  and  reviews,  however,  are  the  same 
for  simple,  short  studies  as  they  are  for  the  most  involved  complex,  multi-year  studies.  They  are: 

•  To  communicate  study  plans,  progress,  issues,  and  risks  to  the  customer  and  other 
stakeholders 

•  To  make  sure  that  the  customer  and  other  stakeholders  buy  into  the  process,  and  by 
extension,  the  results 

•  To  make  decisions  on  eliminating  options  (in  Set  Based  Design  this  would  entail  a 
reduction  of  the  design  space)  or  selecting  an  option  for  the  next  study  activity. 

•  To  aide  in  the  free  flow  of  information  among  team  members,  the  customer,  and  other 
stakeholders 

•  To  solicit  ideas  and  feedback  from  outside  the  team 
Typical  project  reviews  for  a  4  month  to  yearlong  study  are: 

•  Kickoff  meeting 

•  Midpoint  review 

•  90%  review 

Shorter  studies  may  not  have  a  midpoint  review,  while  longer  studies  may  have  additional 
reviews.  The  kickoff  meeting  generally  is  held  when  the  study  guide  is  finalized  or  nearly 
finalized  and  the  study  team  is  in  place.  The  90%  review  is  generally  held  when  the  Work 
Execution  &  Integration  is  almost  complete,  the  final  report  is  nearing  completion,  and  study 
closeout  tasking  is  about  to  commence. 

3.5  Report  Development 

Performing  a  great  study  without  properly  documenting  it  is  a  terrible  waste  of  resources.  The 
results  will  be  suspect  because  the  assumptions  and  methodology  will  be  unclear,  and  any  lesson 
learned  will  be  quickly  forgotten.  A  well  documented  study  will  satisfy  the  Customer  by 
explicitly  answering  important  questions,  will  serve  as  powerful  evidence  to  support 
requirements  development,  will  aide  future  studies  by  obviating  the  need  to  redo  your  work,  and 
will  provide  input  into  design  and  analysis  tool  development. 

Write  a  formal,  serialized  report.  Shorter  studies  may  employ  a  letter  report  style  where  the 
complete  contents  of  the  report  are  contained  in  a  memorandum  from  the  study  leader  to  the 
customer.  More  complex  studies  should  create  a  formal  technical  report.  Guidance  for  the 
format  and  structure  of  a  formal  technical  report  is  contained  in  ANSI/NISO  Z39. 18-2005.  Each 
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formal  report  must  have  a  title  page,  signature  page,  and  report  documentation  page  (Standard 
Form  298). 

The  title  page  should  contain  the  report  title,  report  date,  organizational  logo(s)  of  those 
individuals  on  the  signature  page,  organizational  address  of  the  final  approval  authority, 
distribution  statement,  and  classification  statement.  Guidance  for  distribution  statements  can  be 
found  in  DoDD  5230.24.  Whenever  possible,  a  graphic  image  of  the  design  will  be  prominently 
displayed  on  the  title  page. 

Reports  discussing  technologies  covered  under  "The  International  Traffic  in  Arms  Regulations" 
(Code  of  Federal  Regulation;  22  CFR  Ch.  I,  part  120-130)  must  also  have  the  following 
statement  on  the  title  page: 

"WARNING  -  This  document  contains  technical  data  whose  export  is  restricted  by  the  Arms 
Export  Control  Act  (Title  22,  U.S.C.  Sec.  2751  et  seq.)  or  the  Export  Administration  Act  of 
1979,  as  amended,  Title  50,  U.S.C.,  App  2401,  et  seq.  Violations  of  these  export  laws  are  subject 
to  severe  criminal  penalties.  Disseminate  per  the  provisions  of  OPNAVINST  5510.161." 

The  executive  summary,  or  lead  paragraph  for  a  letter  report,  must  delineate  the  questions  that 
the  customer  seeks  to  have  answered,  explain  why  the  question  is  important,  and  summarize  the 
findings  in  terms  of  the  specific  insights  gained. 

Ensure  the  report  includes  the  following: 

•  Identify  the  customer  and  the  problem  statement 

•  Articulate  the  subject  context  and  importance 

•  Demonstrate  that  the  study  results  answer  the  problem  statement 

•  List  key  assumptions 

•  Identify  tools  and  methodology  used 

Incorporating  the  following  into  the  report  is  recommended: 

•  Highlight  technical  and  other  programmatic  risks 

•  Describe  insights  gained  and  recommendations  for  further  study 

•  Assess  risks  of  study  methodology 

Incorporate  the  study  guide  either  as  an  appendix,  or  as  a  reference. 

If  there  is  a  substantial  -minority  opinion”  it  may  be  appropriate  to  capture  it  in  the  report. 
Distribute  the  report  to  customers  and  stakeholders  as  appropriate  via  official  memo. 

The  study  guide  should  describe  the  contents  of  the  final  report.  The  study  leader  must  be 
mindful  of  scope  growth  in  the  final  report.  While  the  format  and  structure  of  the  final  report 
will  likely  evolve  during  the  study,  the  report  should  remain  focused  on  answering  the  study 
questions.  If  time  and  resources  are  available,  consideration  can  be  made  to  documenting  other 
tangential  topics  uncovered  during  the  course  of  the  study. 

Work  on  the  report  should  start  immediately  following  the  kickoff  meeting.  Progress  on  the 
report  should  be  reported  at  all  subsequent  project  reviews. 
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3.6  Study  Closeout 

3.6.1  Formal  Review  of  the  Technical  Study  Report 

Every  technical  study  report  should  be  reviewed  by  the  appropriate  stakeholders.  The  reputation 
of  the  entire  organization,  not  just  the  study  leader  or  study  participants,  is  impacted  by  the 
quality  of  the  study  report.  The  stakeholder  review  should  ensure  that  all  findings  are  supported 
by  the  evidence  within  the  report  and  statements  of  fact  are  supported  by  appropriate  references. 

3.6.2  Presenting  to  the  Customer 

Present  the  results  of  the  study  to  the  customer.  Concise,  4-page  reports  are  often  more  effective 
than  PowerPoint  presentations  at  quickly  and  reliably  transmitting  essential  quantitative  information 
for  the  purpose  of  making  sound  technical  decisions.  The  use  of  this  fonnat  (the  -Tufte-style”  paper) 
is  encouraged.  Tufte-style  papers  are  formatted  like  a  published  paper  but  are  restricted  to  four 
pages,  printed  out  on  one  11x17  sheet  and  then  folded  in  half.  Elements  include  title,  author, 
executive  summary,  text,  and  illustrations.  Acknowledgements  and  references  can  be  included  as 
needed.  For  more  information  on  Tufte-style  papers  see  (Tufte  2003). 

If  a  PowerPoint  presentation  is  used,  follow  the  presentation  guidance  detailed  in  Appendix  A. 

3.6.3  Disposition  of  and  Archiving  Data 

Upon  completion  of  a  study,  retained  project  data  should  be  put  in  a  long-term  accessible  electronic 
format  on  long-term  storage/retrievable  electronic  media.  Consideration  should  be  made  to 
submitting  technical  reports  to  the  Defense  Technical  Information  Center  (DTIC). 

Proper  disposition  of  project  data  is  a  study  leader  responsibility.  Proper  disposition  will  assure 
the  usefulness  of  these  data  so  future  projects  can  profit  by  lessons  learned  and  make  use  of 
technical  work  already  performed. 

The  data  to  be  disposed  of  by  a  project  generally  fall  into  the  following  categories: 

•  Project  required  deliverables. 

•  Technical  data  developed  to  support  deliverables. 

•  Reference  matter  acquired  for  project  use. 

•  Management  reports  such  as  fiscal  reports  and  progress  reports. 

The  decision  regarding  what  data  are  to  be  retained  after  completion  of  a  project  depends  on  the 
potential  need  or  benefit  expected  for  use  of  these  data  as 

•  Reference  use  to  answer  questions  which  may  arise  during  follow  on  studies, 

•  Use  in  current/future  studies, 

•  Use  in  reproducing  results, 

•  Use  in  presentations,  or 

•  Of  interest  to  future  historians. 

Consideration  should  be  given  to  creating  -Memorandums  for  the  Record”  to  capture  work  on 
dead-ends  and  non-selected  options  that  were  not  included  in  other  formal  reports.  This  work 
can  be  equally  as  useful  in  future  studies  as  the  work  in  the  final  report. 
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During  the  course  of  a  project,  and  particularly  at  the  completion  of  a  project,  data  should  be 
reviewed  and  a  decision  made  as  to  retention  or  disposition.  The  following  steps  are  provided  as 
guidance  to  retaining  sufficient  data  to  meet  the  potential  needs  identified  above  while  keeping 
the  quantity  of  data  retained  to  a  minimum. 

a.  Review  holdings  against  deliverable  requirements  and  fill  deficiencies. 

b.  Purge  the  file  of  insignificant  data,  e.g.,  data  not  required  to  meet  the  potential  needs 
identified  above 

c.  Purge  the  file  of  all  preliminary  documents  for  which  approved  finals  are  on  file. 

d.  Purge  the  file  of  documents  where  the  data  are  contained  in  other  summary  documents 

e.  Segregate,  identify,  and  purge  duplicates. 

3.6.4  Lessons  Learned 

Particularly  for  longer  duration  studies  or  studies  where  new  methods  or  tools  are  used, 
consideration  should  be  given  to  collecting  and  documenting  lessons  learned  in  a  serialized  letter 
or  memorandum.  Where  applicable,  the  lessons  learned  should  provide  recommended  changes 
to  existing  process  documents  or  best  practices  documents.  The  lessons  learned  should  also 
compare  the  actual  cost  and  schedule  to  the  planned  cost  and  schedule  in  the  original  approved 
study  guide. 
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APPENDIX  A  -  PRESENTATION  GUIDELINES 

Title  Slide 

•  Who  gave  the  briefing 

•  The  date  of  the  briefing 

•  Organizational  logo  of  the  presenter 

•  Good  descriptive  title 

•  Who  you  are  giving  the  brief  to  (optional) 

•  Classification  and/or  Distribution  Statement 
All  Follow  on  Slides 

•  Classification  and/or  Distribution  Statement 

•  Date  of  the  briefing  (at  least  month  and  year) 

•  Page  Number 

•  Organizational  logo  of  the  presenter 

•  Last  name  of  presenter  (optional) 

First  Slide  (possibly  two):  Bottom  Line  Up  Front  (BLUF) 

Address  the  following: 

•  Is  this  for  infonnation  or  is  a  decision  needed? 

•  What  level  of  participation  is  required  from  the  audience? 

•  What  is  the  issue?  Present  the  Study  Questions. 

•  What  are  the  answers? 

•  If  decisions  are  needed,  what  are  the  decisions  and  the  decision  options? 

•  What  is  the  urgency  of  the  decision?  What  is  the  impact  of  not  making  a  decision? 

Outline  Slide 

•  Provide  an  outline  for  the  remainder  of  the  presentation. 

Follow  On  Slides 

•  Treat  each  slide  as  a  paragraph  -  one  distinct  message  per  slide.  If  a  slide  is  extremely  busy,  it 
is  likely  because  the  slide  is  attempting  to  convey  multiple  messages. 

•  Because  the  slides  will  be  used  as  a  reference  by  the  audience  after  the  presentation,  the 
Infonnation  needed  to  convey  that  one  message  should  be  on  the  slide.  Slides  should  be  simple, 
but  not  lacking  in  the  required  information.  Overly  abbreviating  the  message  can  lead  to 
ambiguity  and  a  failure  to  accurately  convey  information  to  the  audience. 

•  Clearly  identify  the  message  for  each  slide  -  possibly  use  bumper  stickers. 
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•  Follow  a  logical  progression  of  messages  for  the  succeeding  slides  to  support  the  conclusions 
or  provide  the  necessary  knowledge  for  making  a  decision. 

Provide  sufficient  study  background  and  insight  from  the  study. 

•  Use  images  instead  of  words  where  possible. 

•  Only  include  infonnation  needed  to  convey  the  message  -  eliminate  distracting  details. 

•  Although  summarizing  the  messages  of  previous  slides  is  acceptable;  do  not  repeat  supporting 
information  unless  you  have  a  good  reason. 

Decision  Slides 

•  If  a  decision  is  needed,  include  a  decision  slide  in  the  appropriate  place  in  the  presentation. 

•  The  decision  slide  should  clearly  articulate  the  decision  needed  and  provide  check  boxes  next 
to  the  different  options. 

•  During  the  presentation,  consider  checking  the  box  for  the  decision  made  and  have  a  decision 
official  sign  the  slide. 

Closing  the  Presentation 

•  Include  a  summary  -  restate  study  questions  and  the  answers. 

•  If  a  decision  presentation,  review  the  decision  made. 

•  If  applicable,  include  next  steps 

Notes 

•  The  reasons  for  putting  the  answer  up  front  include: 

-  Many  times  you  will  only  have  a  fraction  of  the  time  you  originally  thought  you  had.  If  you 
get  the  answer  out  at  the  start,  you  make  sure  you  have  conveyed  your  message. 

-  By  knowing  the  answer,  the  audience  has  the  framework  to  interpret  the  rest  of  the  slides. 
Hopefully,  you  will  stop  the  tendency  of  the  audience  to  wonder  where  the  presentation  is 
leading  them.  Many  times,  your  audience  will  focus  on  a  specific  aspect  of  the  solution;  this 
gives  you  the  opportunity  to  jump  to  the  appropriate  slide  in  the  presentation  (or  even  to  a 
backup  slide  if  necessary). 

•  Leave  out  data  that  does  not  directly  address  the  issue  at  hand.  Many  engineers  structure  their 
presentation  like  a  photo-album;  they  present  their  journey  through  the  study.  Unfortunately, 
while  this  may  be  interesting  to  the  presenter,  it=snot  effective  in  rapidly  conveying  information. 
Only  that  infonnation  which  is  central  to  the  conclusions  made  by  the  presentation  should  be 
incorporated.  The  presentation  should  be  structured  around  the  conclusions,  not  the  process  used 
to  develop  the  conclusions.  In  other  words,  the  presentation  should  be  like  a  story,  but  the  plot 
of  the  story  should  be  the  topic  itself,  not  the  analysis  of  the  topic. 

•  Be  honest.  DO  NOT  suppress  information  that  appears  to  run  counter  to  your  conclusions. 
Rather,  show  the  infonnation  and  give  the  reasons  why  you  believe  the  infonnation  does  not 
invalidate  your  conclusions. 

•  Leave  out  data  of  questionable  quality  that  support  the  conclusion  of  the  study.  If 
appropriate,  include  the  data  of  questionable  quality  as  well  as  a  risk  assessment  if  the  data 
conflicts  with  conclusion  of  the  study. 
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•  Never  use  data  that  you  do  not  understand  or  cannot  clearly  explain  to  support  the  conclusion 
of  the  study.  This  data  can  be  used  in  presenting  a  risk  assessment  of  the  validity  of  the  study 
results. 

•  Do  not  read  the  slides,  tell  the  story.  The  slides  should  tell  the  story  on  their  own,  but  effective 
visual  communication  is  very  different  from  effective  oral  communication.  Use  the  two  modes 
to  complement  each  other,  not  to  repeat  each  other. 

•  Use  page  numbers  on  all  slides.  For  configuration  management  reasons,  also  consider  putting 
the  date  and  filename  on  all  slides.  Adding  organizational  logos  to  every  slide  further  helps 
ensure  that  the  originator  of  a  hard  copy  slide  that=s  been  faxed  multiple  times  can  be 
detennined. 

•  Have  a  good  understanding  of  the  accuracy  of  your  results. 

•  When  comparing  results  on  options,  focus  on  statistical  significance  of  the  differences.  If 
numbers  are  presented,  ensure  precision  of  display  does  not  exceed  the  accuracy  of  the  metric. 

Do  not  claim  one  option  is  better  than  another  if  the  differences  in  the  metrics  are  not  statistically 
significant. 

•  Provide  interpretations  of  results.  What  generalized  lessons  can  be  learned?  Often  we  are  not 
concerned  with  the  particular  details  of  the  concept  studied;  rather  we  are  interested  on  what  is 
learned  to  impact  decisions  at  hand.  Are  the  results  a  function  of  the  details  or  can  they  provide 
the  generalized  answers? 

•  Do  not  use  too  many  slides.  Use  the  fewest  necessary  to  make  your  point.  In  no  case  should 
you  have  more  than  1  slide  per  minute  of  presentation.  Ideally,  the  ratio  should  be  about  1  slide 
per  2  to  3  minutes  -  it  allows  you  to  tell  the  story  and  allows  the  story  to  be  heard  instead  of  just 
getting  through  the  slides. 

•  Strive  to  have  an  engineering  level  of  detail  about  an  order  of  magnitude  greater  than  that 
which  is  presented.  This  helps  ensure  that  what  you  present  has  a  solid  foundation,  is  not  just  a 
fantasy,  and  can  likely  remain  accurate  even  if  one  of  the  lower  level  details  is  found  to  be 
inaccurate.  The  increased  level  of  detail  could  be  included  in  backup  charts,  should  questions 
come  up. 

•  Reference  other  sources  that  you  use  data  from. 
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